Automated assistant suggestions for third-party vehicle computing devices with restricted architecture

ABSTRACT

Implementations set forth herein relate an automated assistant that provides suggestion elements at an OEM application and/or third-party application that is accessible via an OEM vehicle computing device, which employs restrictions on communications between local applications. The automated assistant can render suggestions via OEM vehicle applications and/or third-party applications, and initialize actions for performance via the OEM vehicle applications and/or the third-party applications—despite the limitations. The suggestions can be rendered at a graphical user interface of a vehicle, and include content that, when spoken by a user, causes the automated assistant to initialize such actions. Actions that are suggested by the automated assistant can be performed by third-party applications that are different from a third-party application that is rendering the suggestion. In this way, a common automated assistant can be indiscriminately available via various vehicle computing devices, despite differences in restrictions of communications between vehicle applications.

BACKGROUND

Computing devices and their respective applications typically providerobust functionality to assist a user with various tasks. A provider ofa particular application or device may not expect the user to employ allfunctionality during an initial use, but rather learn the functions ofthe application or device over time. When a particular device providesaccess to an automated assistant, the automated assistant can act tofamiliarize the user with various functions using suggestions, whichdirect the user to control one or more actions of the particular deviceand/or application. However, in some instances, such as at a vehiclecomputing device or other OEM computing device, the automated assistantmay be accessible through the vehicle computing device but limited inthe types of data that can be gathered from operations of the vehiclecomputing device. As a result, any suggestions made by the automatedassistant may not be suitably tailored to functions of the vehiclecomputing device and/or other applications that may enable moreeffective and/or efficient use of the vehicle and/or vehicle computingdevice.

Furthermore, such limitations can prevent the user from learning aboutsafety functions of the vehicle computing device sooner. In other words,because the vehicle computing device can provide a plethora of safetyfeatures that a user could not reasonably be expected to instantlyacknowledge, any delays in learning about such features could limit thesafety and/or utility of the vehicle. Additionally, because a user mayspend much of their time controlling the vehicle in order to navigaterather than learning features, the user would not be reaping thebenefits, such as reducing power consumption and increasing efficiency.This can be especially apparent when the vehicle computing device is notexclusively proactive about suggesting functionality to the user—as suchbenefits are often realized when users outsource various tasks to acomputing device.

SUMMARY

Implementations set forth herein relate to a hierarchy of vehicleapplications provided at a vehicle computing device that also includesan automated assistant, which can render suggestions via the vehicleapplications and initialize actions via the vehicle applications. Thevehicle applications can be loaded onto a vehicle computing device of avehicle, and can provide a variety of functionality related tonavigation, communication, media, and/or any other type of servicescapable of being provided by a computing device. The vehicle computingdevice can operate according to an operating system that is loaded ontothe vehicle computing device and includes one or more original equipmentmanufacturer (OEM) applications. The OEM applications can be provided byan entity or source that also provides the vehicle. Furthermore, thevehicle computing device can also include one or more vehicleapplications provided by a third-party source (i.e. third partyapplications rather than native vehicle applications), as well as anautomated assistant that is provided by a separate source from thethird-party source. Communications between the automated assistant andthe other applications can be restricted to communications which occurvia the vehicle computing device, which can improve reliability of thevehicle systems. The automated assistant can be remote from the vehiclecomputing device. One or more actions suggested by the automatedassistant may be performed by the local vehicle applications (whethernative OEM applications or third-party applications), which can improvereliability of vehicle performance, particularly in situations where itis desirable that a vehicle retains full control of its systems, such asin an autonomous vehicle environment. One or more actions may beperformed by the remote automated assistant, which can protect user dataand information associated with the automated assistant (assistantdata), and thus improve user privacy and data security, by preventingthe local vehicle applications from having access to the user data.

In some implementations, while the user is riding in the vehicle, an OEMvehicle application and/or the automated assistant can render agraphical user interface that includes a variety of different contentthat is based on data from one or more other vehicle applications. Forexample, the graphical user interface can include a first area forrendering navigation related content, a second area for renderingcommunication related content, and/or a third area for renderingautomated assistant content. In order for a user to learn variousfunctions of the vehicle computing device and/or the vehicleapplications, the automated assistant can provide and/or causesuggestions to be rendered at each area of the graphical user interface.As a result, despite an automated assistant being accessible via avariety of different vehicles, by using data from one or more othervehicle applications provided at the vehicle computing device theautomated assistant will nonetheless be able to provide suggestions forusers, regardless of the vehicle in which the automated assistant isoperating.

In some implementations, when the graphical user interface is displayingat least the first area and the second area, the OEM vehicle applicationcan generate data characterizing what is provided at the user interface,and provide this generated data to the automated assistant. The data cancharacterize or, as used herein, indicate some or all of the contentprovided at the graphical user interface, in order to provide a contextwith which the automated assistant can generate suggestions. Forexample, when the first area is displaying navigation directions to adestination, and the second area is displaying a notification regardingan incoming phone call, the OEM vehicle application can generate datacharacterizing (or indicating) the status of the incoming phone call andthe status of the navigation to the destination. In someimplementations, the automated assistant can use the data to generateone or more suggestions or actions to be performed by the automatedassistant, and/or any other application that is accessible via thevehicle computing device.

For example, the automated assistant can provide a suggestion element atthe graphical user interface for invoking a messaging application toprovide an estimated a-time of arrival to a person who is the source ofthe incoming call. The automated assistant can cause the suggestionelement to be provided at the second area of the graphical userinterface because the suggestion is related to communications. Thesuggestion element can include natural language content characterizing(or indicating) any action the assistant can take such as, “Send Meganmy estimated time of arrival.” The natural language content can be textprovided at the suggestion element, can be an example utterance forinvoking the automated assistant to perform an action associated withthe suggestion element, and/or can otherwise characterize a spokenutterance for invoking the automated assistant. For example, in order toinvoke the automated assistant to cause a message to be sent, the usercan tap the display panel at a location in which the suggestion elementis being rendered, and/or provide a spoken input that includes at leasta portion of the spoken utterance rendered at the graphical userinterface (e.g., “Assistant, send a message to Megan indicating myETA.”). In another example, the spoken input can have the same meaningor intent as the natural language content of the suggestion element.

In some implementations, the OEM vehicle application can provide theautomated assistant with various data over time, at least in response tochanges in content being rendered at the display panel of the vehiclecomputing device. As subsequent data is received by the automatedassistant from the OEM vehicle application, the automated assistant cangenerate further suggestions, and/or rank suggestions according to avariety of different properties. In some instances, a suggestion elementprovided by the automated assistant can be ranked according to astrength of correlation between assistant data and vehicle applicationdata provided by the OEM vehicle application. For example, vehicleapplication data characterizing a destination displayed at the displaypanel can have a stronger correlation to assistant data that alsoidentifies the destination, and this “stronger” correlation can berelative to when the vehicle application data does not identify thedestination that is also identified by the assistant data.

A set of suggestion elements can be generated for each area of thegraphical user interface that is associated with a particular type ofapplication and/or activity. For each set of suggestion elements, a rankcan be assigned for each suggestion element of the set of suggestionelements. In this way, when a suggestion element is to be rendered at aparticular area of the graphical user interface, a suggestion elementthat is selected to be rendered can be a highest ranking suggestionelement from the set of suggestion elements corresponding to theparticular area. In some implementations, a ranking for a particularsuggestion element can be based on content of the entire graphical userinterface, previous interactions between the user and the automatedassistant, third-party application data that is available to theautomated assistant with permission from the user, vehicle datacharacterizing one or more operations of the vehicle, and/or any otherinformation from which a ranking for a suggestion can be based.

In some implementations, the automated assistant can provide suggestionsfor third-party vehicle applications to render, and these suggestionscan be generated based on information from the OEM vehicle application.For example, multiple different third-party applications can beaccessible via the vehicle computing device, and can communicate withthe OEM vehicle application in order to send and/or receive data. Insome implementations, the OEM vehicle application can limitcommunications between other applications on the vehicle computingdevice. Therefore, the third-party application may not be able tocommunicate with the automated assistant locally, but rather communicateand exclusively via vehicle computing device—rather, the automatedassistant can thus be aware of operations being performed by thethird-party application using data that is provided by the OEM vehicleapplication instead of through direct communication with the third-partyapplication.

For example, when the third-party application is rendering content atthe display panel of the vehicle computing device, the OEM vehicleapplication can generate data that characterizes the content, andprovide the data to the automated assistant. Alternatively, oradditionally, the third-party application can communicate with a remotecomputing device, such as a server, in order to provide data to theautomated assistant (e.g., using an automated assistant applicationprogramming interface (API)). Suggestion data can then be generated viathe automated assistant, and then shared with the OEM vehicleapplication and/or a remote device associated with the third-partyapplication. One or more suggestion elements can then be presented atthe graphical user interface of the third-party application based on thesuggestion data. In this way, despite the OEM vehicle applicationlimiting communications between local applications to communicationswhich occur exclusively via the vehicle computing device, the automatedassistant can nonetheless provide suggestions to the user for variousactions that the user may not be aware of, at least with respect to thevehicle computing device and/or the vehicle.

In some implementations, the automated assistant can bypass providingsuggestions related to actions that have recently been performed at thedirection of the user via a third-party application and/or the OEMvehicle application. For example, when a third-party applicationcorresponds to a vehicle maintenance application, and the vehiclemaintenance application provides a notification indicating that a partof the vehicle needs attention, the automated assistant can be informedof this notification via the OEM vehicle application. For instance, whenthe vehicle maintenance application indicates that the vehicle needs gasand/or charge, the vehicle maintenance application can providenotifications about nearby places to refuel the vehicle. The OEM vehicleapplication can generate data based on the notifications and/or contentbeing rendered at the graphical user interface of the third-partyapplication, and provide the generated data to the automated assistant.The automated assistant can use the data to generate suggestions, whichcan be ranked and/or filtered according to what has already beenpresented to the user.

For example, when a generated suggestion corresponds to a nearby placeto refuel the vehicle, the generated suggestion can be ranked (i.e.,prioritized) lower than a separate suggestion that does not correspondto the nearby place to refuel the vehicle. As a result, othersuggestions regarding, for example, spoken utterances for obtainingother information about the vehicle (e.g., “Assistant, what is myhighway miles per gallon?”), can be prioritized higher than suggestionsrelated to nearby places to refuel. Alternatively, or additionally,other suggestions can be generated based on a comparison between thedata that is based on the content from the third-party application, andassistant data, where assistant data is data that is associated withinteractions between the user and the automated assistant. For example,the assistant data can comprise content of previous interactions betweenthe user and the automated assistant, contact information or calendarinformation linked to, or associated with, an account of the user andinteracted with by the user through the automated assistant, and/or theassistant data can comprise time and/or location information associatedwith a time and/or location of user interaction with the automatedassistant. In one example, the data from the OEM vehicle application canbe compared to the assistant data to determine that the notificationfrom the third-party application is associated with an operating featureof the vehicle. Based on this determination, the automated assistant candetermine when the user has previously participated in a dialoguesession with the automated assistant regarding operating features of thevehicle.

For instance, the user may have queried the automated assistant to findout what the appropriate tire pressure is for their vehicle. Therefore,in response to receiving the data from the OEM vehicle application, theautomated assistant can generate a suggestion characterizing a spokenutterance such as, “Assistant, what is the tire pressure of my vehicle?”Suggestion data corresponding to this suggestion can be transmitted fromthe automated assistant to the OEM vehicle application and/orthird-party application, and the third-party application can thenpresent the suggestion at the graphical user interface with the contentcharacterizing the fuel status of the vehicle. In this way, while theuser is being notified about refueling their vehicle, the user can learnto conduct similar dialogue sessions to check on other matters relatedto the vehicle, in order that such matters can be addressed sooner,thereby promoting a healthy routine of vehicle maintenance.

Other implementations may include a computer program or non-transitorycomputer readable storage medium storing instructions executable by oneor more processors (e.g., central processing unit(s) (CPU(s)), graphicsprocessing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s))to perform a method such as one or more of the methods described aboveand/or elsewhere herein. Yet other implementations may include a systemof one or more computers and/or one or more robots that include one ormore processors operable to execute stored instructions to perform amethod such as one or more of the methods described above and/orelsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A, FIG. 1B, and FIG. 1C illustrate views of a vehicle thatincludes an OEM vehicle computing device that provides access to anautomated assistant.

FIG. 2A, FIG. 2B, and FIG. 2C illustrate one or more implementations ofan automated assistant that provides suggestion elements at an OEMapplication and/or third-party application that is accessible via an OEMvehicle computing device, which employs restrictions on communicationsbetween local applications.

FIG. 3 illustrates a system that provides an OEM vehicle computingdevice that includes a vehicle automated assistant, which can rendersuggestions via OEM vehicle applications and/or third-partyapplications, and initialize actions via the OEM vehicle applicationsand/or the third-party applications.

FIG. 4 illustrates a method for providing one or more suggestionelements at a graphical user interface of a third-party vehiclecomputing device and/or a third-party application operating at thethird-party vehicle computing device.

FIG. 5 is a block diagram of an example computer system.

DETAILED DESCRIPTION

FIG. 1A, FIG. 1B, and FIG. 1C illustrate view 100, view 130, and view140 of a vehicle 108 that includes an OEM vehicle computing device 104that provides access to an automated assistant 128, which can causesuggestions to be rendered at a graphical user interface 106 of an OEMvehicle application 124 and/or a third-party application 126.Specifically FIG. 1A illustrates a view 100 of the OEM vehicle computingdevice 104 providing the graphical user interface 106 with one or moreat different areas corresponding to different applications and/orcontent that can be presented at the graphical user interface 106. TheOEM vehicle computing device 104 can include one or more OEM vehicleapplications 124, one or more third-party applications 126, and/or anautomated assistant 128. While the user 102 is riding in the vehicle108, one or more applications accessible via the OEM vehicle computingdevice 104 can communicate with a particular OEM vehicle application 124in order to cause certain content to be rendered at the graphical userinterface 106.

For example, the OEM vehicle computing device 104 can include an OEMvehicle application 124 such as a navigation application, which cancause content to be rendered at a first area 110 of the graphical userinterface 106. Alternatively, or additionally, the OEM vehicle computingdevice 104 can include a third-party application 126 such as a messagingapplication, which can cause other content to be rendered at a secondarea 114 of the graphical user interface 106. Each application thatprovides data for rendering content at the graphical user interface 106can communicate with a particular OEM vehicle application 124. Thisparticular OEM vehicle application can communicate with other OEMvehicle applications 124, third-party applications 126, and theautomated assistant 128. For example, the navigation application canprovide data characterizing upcoming navigation instructions, which canbe rendered as a first graphical element 112 in the first area 110 ofthe graphical user interface 106. Furthermore, the messaging applicationcan provide other data characterizing (or indicating content of) anincoming message, and the other day that can be rendered as a secondgraphical element 116 by the particular OEM vehicle application 124 atthe graphical user interface 106.

In some implementations, the particular OEM vehicle application 124 canmediate communications between applications by generating data thatcharacterizes content provided by one or more applications of the OEMvehicle computing device 104, and providing the generated data to theautomated assistant 128. For example, the generated data cancharacterize natural language content provided in the first graphicalelement 112 and other natural language content provided in the secondgraphical element 116. The particular OEM vehicle application 124 canprovide this generated data to the automated assistant 128 oh, and theautomated assistant 128 can use the generated data to provide additionalsuggestions for each respective area of the graphical user interface106.

FIG. 1B illustrates a view 130 of the graphical user interface 106providing make space for a suggestion element for each area of thegraphical user interface 106 based on data provided by the automatedassistant 128, and which suggestion element was generated based on datafrom a particular OEM vehicle application 124 and based on the data fromthe OEM vehicle application 124 being associated with assistant data.The vehicle application data and the assistant data can be associated,or correlated, if the vehicle application data corresponds at least inpart to data which is associated with interactions between the user andthe automated assistant. For example, the vehicle application data maycorrespond to content of a previous interaction or dialog sessionbetween the user and the automated assistant, relate to a previousaction performed by the automated assistant, indicate an actionassociated with a location or contact linked to the user, or otherwisecorrespond to the assistant data. Specifically, a particular OEM vehicleapplication 124 can provide the automated assistant 128 with datacharacterizing content of the first graphical element 112. The automatedassistant 128 can receive the data from the OEM vehicle application 124and generate a suggestion data based on the received data. In someimplementations, the received data can be compared to one or moreactions capable of being performed by the automated assistant 128, andbased on the comparison, suggestion data can be generated. Additionally,or alternatively, the OEM vehicle application 124 can provide theautomated assistant 128 with other data characterizing content of thesecond graphical element 116. The automated assistant 128 can otherreceived the data to be processed for determining a correlation betweenthe other received data and one or more actions capable of beingperformed by the automated assistant 128. Based on the comparison, othersuggestion data can be generated and provided to the OEM vehicleapplication 124.

In response to the OEM vehicle application 124 receiving suggestion datafrom the automated assistant 128, the OEM vehicle application 124 cancause suggestion elements to be rendered at the graphical user interface106. For example, the OEM vehicle applications 124 can include alauncher application that renders contact characterizing operationalstatus of one or more applications, and also initialize one or moreactions of one or more applications according to data received from theautomated assistant 128. For example, the automated assistant 128 canprovide suggestion data characterizing a first suggestion element 132,which can characterize a spoken utterance that, when spoken by the userand/or another user, causes the automated assistant to initialize orperform a particular action. For example, the vehicle application canprovide an indication of the selection to the automated assistant, andthe automated assistant can cause another application of the vehiclecomputing device to perform the action. The other application can be athird-party application (i.e. from a different source than the automatedassistant and the vehicle OEM applications) or one of the OEM vehicleapplications. Furthermore, the automated assistant 128 can provide othersuggestion data characterizing a second suggestion element 134, whichcan characterize a different spoken utterance that, when is spoken bythe user and/or another user, causes the automated assistant 128 toinitialize or perform a different particular action. For instance, thefirst suggestion element 132 can characterize a spoken utterance suchas, “Assistant, what is my ETA?”

In response to the user tapping on the suggestion element at a displaypanel 120, or providing a spoken input that includes at least someamount of content of the suggestion element 132, the automated assistantcan communicate an estimated time of arrival at the destination to theuser. Furthermore, the second suggestion element 134 can characterize aspoken utterance such as, “Assistant, reply to Jane.” In response to theuser tapping the second suggestion element 134 at the display panel 120,and/or providing a spoken input that includes at least some of thecontent of the second suggestion element 134, the automated assistant128 can prompt the user to provide additional content for a message tobe transmitted to a contact named Jane. In this way, the user can be puton notice that they can employ the automated assistant to initializeactions of the OEM vehicle applications 124 and/or the third-partyapplications 126, and/or otherwise perform actions based on informationprovided by the vehicle application 124 and/or the third-partyapplications 126.

FIG. 1C illustrates a view 140 of the automated assistant 128 causing alauncher OEM vehicle application 124 to provide a suggestion elementthat is based on content presented at the graphical user interface 106for multiple different applications. Specifically, a third suggestionelement 142 can be generated by the automated assistant 128 usingvehicle application data provided by the launcher OEM vehicleapplication 124. The launcher OEM vehicle application 124 can receivedata from multiple different third-party applications and/or other OEMvehicle applications 124 for rendering at the graphical user interface106. The launcher OEM vehicle application 124 can generate datacharacterizing content being presented at the graphical user interface106, and provide the generated data to the automated assistant 128. Theautomated assistant 128 can use received data from the launcher OEMvehicle application 124 to generate suggestion data, which can berendered in a third area 118 of the graphical user interface 106.

The suggestion data can be generated via the automated assistant 128 inorder to provide a suggestion to the user that relates to multipledifferent sources of content being presented at the graphical userinterface 106. In some implementations, the generated data from thelauncher OEM vehicle application 124 can characterize an ongoing actionbeing performed by a separate OEM vehicle application, as well as arecently performed action or recently presented notification provided bya separate third-party application 126. In order to generate a suitablesuggestion for the user 102, the automated assistant 128 can determinewhether any of the generated data (which characterizes or indicates thecontent being presented at the graphical user interface 106) carries anyrelationship to assistant data that is accessible to the automatedassistant 128. For example, because of the automated assistant 128 canbe linked to an account of the user 102, the automated assistant 128 canbe notified of the incoming message to be third-party application from acontact of the user called “Jane.” Based on this association orrelationship between the content of the second graphical element 116 andthe assistant data that also indicates the incoming message, theautomated assistant 128 can generate a blank message for replying toJane. Furthermore, the automated assistant 128 can also determine arelationship between content of the first graphical element 112 and theassistant data. For example, the assistant data can characterize one ormore actions capable of being performed or initialized by the automatedassistant 128.

The suggestion data can be generated via the automated assistant 128 inorder to provide a suggestion to the user that relates to multipledifferent sources of content being presented at the graphical userinterface 106. In some implementations, the generated data from thelauncher OEM vehicle application 124 can characterize an ongoing actionbeing performed by a separate OEM vehicle application, as well as arecently performed action or recently presented notification provided bya separate third-party application 126. In order to generate a suitablesuggestion for the user 102, the automated assistant 128 can determinewhether any of the generated data carries any relationship to assistantdata that is accessible to the automated assistant 128. For example,because of the automated assistant 128 can be linked to an account ofthe user 102, the automated assistant 128 can be notified of theincoming message to be third-party application from the user's knowncontact “Jane.” Based on this relationship between the content of thesecond graphical element 116 and the assistant data that also indicatesthe incoming message, the automated assistant 128 can generate a blankmessage for replying to Jane. Furthermore, the automated assistant 128can also determine a relationship between content of the first graphicalelement 112 and the assistant data. For example, the assistant data cancharacterize one or more actions capable of being performed orinitialized by the automated assistant 128.

FIG. 2A, FIG. 2B, and FIG. 2C illustrate one or more implementations ofan automated assistant 228 that provides suggestion elements at an OEMapplication and/or third-party application that is accessible via an OEMvehicle computing device 204, which employs restrictions oncommunications between local applications. For instance, as provided inview 200 of FIG. 2A, the OEM vehicle computing device 204 can includeOEM vehicle applications 224, third-party applications 226, and/or anautomated assistant 228. The OEM vehicle computing device 204 canenforce access restrictions on the automated assistant 228, therebylimiting access to locally stored data of the OEM vehicle applications224 and locally stored data of the third-party applications 226 to theautomated assistant 228. However, one or more of the OEM vehicleapplication 224 can provide some amount of data to the automatedassistant 228 regarding operations of a vehicle 208, the OEM vehiclecomputing device 204, the OEM vehicle applications 224, the third-partyapplications 226, and/or any other information that can be associatedwith the vehicle 208.

For example, a launcher OEM vehicle application can generate contentdata characterizing content of a graphical user interface 206 beingrendered at a display panel 220 of the OEM vehicle computing device 204.The content data can characterize natural language content presented ina first area 210 of the graphical user interface 206. The first area 210can include a first graphical element 212 associated with a third-partyapplication, such as a navigation application. The content data can alsocharacterize natural language content presented in a second area 214 ofthe graphical user interface 206. The second area 214 can include asecond a graphical element 216 associated with a third-partyapplication, such as a messaging application.

In order to familiarize a user 202 with functionality that allows theautomated assistant at 228 to initialize OEM vehicle applications 224and/or third-party applications 226, the automated assistant 228 canprovide suggestion elements related to such functionality. In otherwords, the automated assistant 228 can use the content data from the OEMvehicle application 224 to generate suggestions for spoken utterancesthat can be provided by the user 202 for invoking the automatedassistant 228 to initialize performance of an action by another OEMvehicle application 224 and/or a third-party application 226.

For example, the automated assistant 228 can use the content data togenerate suggestion data corresponding to a suggestion for the user toprovide a command that relates to both the content of the firstgraphical element 212 and content of the second graphical element 216.In this way, the automated assistant 228 is acting within the bounds ofthe restrictions enforced by the OEM vehicle computing device 204 whilestill providing useful suggestions to promote usage of all functionalityof the OEM vehicle computing device 204. For instance, the suggestiondata can characterize a spoken utterance such as, “Assistant, where isthe nearest grocery store?” This is spoken utterance can be suggestedbased on the navigation application conducting an ongoing action ofdirecting the user to a particular destination, and the user 202receiving a message regarding picking up coffee during their drive inthe vehicle 208. The suggestion data can be provided to the launcher OEMvehicle application, which can process the suggestion data and cause athird graphical element 214 to be presented within a third area 222 ofthe graphical user interface 206. As a result, a suggestion is presentedto the user 202 without the automated assistant 228 violating any accessrestrictions set forth by the OEM vehicle computing device 204, therebyallowing other instances of the automated assistant to operate on othervehicles with similar restrictions. Additionally or alternatively, theautomated assistant 228 can use the content data to generate suggestiondata corresponding to a suggestion for the user to provide a commandthat relates to only one of the content of the first graphical element212 or the content of the second graphical element 216.

In some implementations, in response to the user tapping the thirdgraphical element 216 and/or providing the identified spoken utteranceto the OEM vehicle computing device 204, the automated assistant 228 canbe invoked. Furthermore, in response to the spoken utterance, theautomated assistant at 228 can initialize performance of one or morecorresponding actions by the navigation application. For instance, inresponse to the automated assistant 228 receiving the spoken utterance,“Assistant where is the nearest grocery store?” The automated assistant228 can generate command data to be provided to the launcher OEM vehicleapplication and/or a server device that is associated with thenavigation application. The command data can characterize one or morecommands that have been generated according to an API corresponding tothe navigation application. When the launcher OEM application and/or theserver device receives the command data, the launcher OEM applicationand/or the server device can initialize execution of the one or morecommands via the navigation application, and/or any other third-partyapplication 226.

In response to execution of one or more are the commands beinginitialized, the navigation application can cause a third-partyapplication interface 242 to be provided at the graphical user interface206, as illustrated in view 240 of FIG. 2B. Therefore, despite thelimitations set forth by the launcher OEM vehicle application, the usercan provide spoken utterances to the automated assistant 228 in order toinvoke a third-party application to perform one or more actions at theOEM vehicle computing device 204. In furtherance of the aforementionedexample, the third-party application 226 can provide content to thelauncher OEM vehicle application for providing the third-partyapplication interface 242. Furthermore, the content can characterize alocation of a nearby grocery store, details regarding the route throughwhich the user is currently taking, and details about when the user 202will arrive at the destination.

In some implementations, the launcher OEM vehicle application cangenerate data characterizing the content provided at the graphical userinterface 206. This generated data can be provided to the automatedassistant 228, which can generate further suggestions based on recentchanges to the graphical user interface 206. Specifically, the generateddata can characterize content provided at the third-party applicationinterface 242, and the automated assistant 228 can use the generateddata in order to generate other suggestion data for causing theautomated assistant 228 to initialize performance of one or more actionsby an OEM vehicle application 224 and/or a third-party application 226.Furthermore, the suggestion data can characterize one or more actionsthat are relevant to the content provided at the graphical userinterface 206, despite the one or more actions being performed by anapplication that is different from the navigation application, and/orother application that provided the content being presented at thegraphical user interface 206.

As provided in view 260 of FIG. 2C, the automated assistant can causethe launcher OEM vehicle application to provide one or more suggestions262 for invoking a separate application to perform one or more actionsusing a different application. The automated assistant 228 can use datacharacterizing content previously rendered at the graphical userinterface 206, and content currently being rendered at the graphicaluser interface 206, in order to generate a suitable suggestion for theuser 202. For example, because the messaging application was indicatedas having a new message from Jane, in FIG. 2A, and the ETA from thenavigation application has been provided at FIG. 2C, the automatedassistant can suggest that the user invoke the automated assistant tosend this new ETA to Jane. Specifically, a suggestion data generated bythe automated assistant 228 can characterize a spoken utterance such as,“Assistant, send my new ETA to Jane.” The suggestion data can beprovided from the automated assistant 228 to the launcher OEM vehicleapplication and/or a server device, associated with the navigationapplication, in order to cause one or more suggestions 262 to bepresented at the graphical user interface 206 according to thesuggestion data.

FIG. 3 illustrates a system 300 provides an OEM vehicle computing device318 that includes a vehicle automated assistant 322, which can rendersuggestions via OEM vehicle applications 320 and/or third-partyapplications 332, and initialize actions via the OEM vehicleapplications 320 and/or the third-party applications 332. The system 300can include an automated assistant 304, which can operate as part of anassistant application that is provided at one or more computing devices,such as the OEM vehicle computing device 318 and/or a server device 302.A user can interact with the automated assistant 304 via an assistantinterface, which can be a microphone, a camera, a touch screen display,a user interface, and/or any other apparatus capable of providing aninterface between a user and an application.

For instance, a user can initialize the automated assistant 304 byproviding a verbal, textual, and/or a graphical input to an assistantinterface (e.g., an assistant interface of the OEM vehicle computingdevice 318 and/or an assistant interface of any other client device) tocause the automated assistant 304 to perform a function (e.g., providedata, control a peripheral device, access an agent, generate an inputand/or an output, etc.). The OEM vehicle computing device 318 caninclude a display device, which can be a display panel that includes atouch interface for receiving touch inputs and/or gestures for allowinga user to control applications of the OEM vehicle computing device 318via the touch interface. In some implementations, the OEM vehiclecomputing device 318 can lack a display device, thereby providing anaudible user interface output, without providing a graphical userinterface output; in such an arrangement, the vehicle computing device318 may be coupled or connected to an external display device to renderthe graphical user interface described herein. Furthermore, the OEMvehicle computing device 318 can provide a user interface, such as amicrophone(s), for receiving spoken natural language inputs from a user.In some implementations, the OEM vehicle computing device 318 caninclude a touch interface and can be void of, or include, a camera, butcan optionally include one or more other sensors.

The OEM vehicle computing device 318 and/or other third-party serverdevice(s) 350 can be in communication with the server device 302 over anetwork 340, such as the internet. Additionally, the OEM vehiclecomputing device 318 and other client devices can be in communicationwith each other over a local area network (LAN), such as a Wi-Finetwork. The automated assistant 322 can offload computational tasks tothe server device 302 in order to conserve computational resources atthe OEM vehicle computing device 318. For instance, the server device302 can host the automated assistant 304, and the OEM vehicle computingdevice 318 can transmit inputs received at one or more assistantinterfaces to the server device 302. However, in some implementations,the automated assistant 304 can be hosted at the OEM vehicle computingdevice 318 as a client automated assistant 322.

In various implementations, all or less than all aspects of theautomated assistant 304 can be implemented on the OEM vehicle computingdevice 318. In some of those implementations, aspects of the automatedassistant 304 are implemented via the client automated assistant 322 ofthe OEM vehicle computing device 318 and can interface with the serverdevice 302, which can implement other aspects of the automated assistant304. The server device 302 can optionally serve a plurality of users andtheir associated assistant applications via multiple threads. Inimplementations where all or less than all aspects of the automatedassistant 304 are implemented via a client automated assistant 322 atthe OEM vehicle computing device 318, the client automated assistant 322can be an application that is separate from an operating system of theOEM vehicle computing device 318 (e.g., installed “on top” of theoperating system)—or can alternatively be implemented directly by theoperating system of the OEM vehicle computing device 318 (e.g.,considered an application of, but integral with, the operating system).

In some implementations, the automated assistant 304 and/or the clientautomated assistant 322 can include an input processing engine 306,which can employ multiple different modules for processing inputs and/oroutputs for the OEM vehicle computing device 318 and/or the serverdevice 302. For instance, the input processing engine 306 can include aspeech processing engine 308, which can process audio data received atan assistant interface to identify the text embodied in the audio data.The audio data can be transmitted from, for example, the OEM vehiclecomputing device 318 to the server device 302 in order to preservecomputational resources at the OEM vehicle computing device 318.

The process for converting the audio data to text can include a speechrecognition algorithm, which can employ neural networks, and/orstatistical models for identifying groups of audio data corresponding towords or phrases. The text converted from the audio data can be parsedby a data parsing engine 310 and made available to the automatedassistant as textual data that can be used to generate and/or identifycommand phrase(s), intent(s), action(s), slot value(s), and/or any othercontent specified by the user. In some implementations, output dataprovided by the data parsing engine 310 can be provided to a parametermodule 312 to determine whether the user provided an input thatcorresponds to a particular intent, action, and/or routine capable ofbeing performed by the automated assistant 304 and/or an application oragent that is capable of being accessed via the automated assistant 304.For example, assistant data 316 can be stored at the server device 302and/or the OEM vehicle computing device 318 as vehicle data 324, and caninclude data that defines one or more actions capable of being performedby the automated assistant 304 and/or client automated assistant 322, aswell as parameters necessary to perform the actions.

In some implementations, the automated assistant 304 can be accessiblevia an OEM vehicle computing device 318, which can limit localcommunication between applications that are accessible at the OEMvehicle computing device 318. The automated assistant 304 can operate toprovide suggestions for invoking the automated assistant 304 toinitialize an OEM vehicle application 320, a third party application332, and/or a client device to perform one or more actions. Theautomated assistant 304 can be accessible via the OEM vehicle computingdevice 318, which can provide access to an instance of an automatedassistant 322 that is in communication with the server device 302.Additionally, or alternatively, the automated assistant 304 can beprovided at the OEM vehicle computing device 318.

The vehicle computing device 318 can operate an application interactionengine 334 to limit transmission of data between the automated assistant322 and the third-party applications 332, and/or between the automatedassistant 322 and OEM vehicle applications 320. The applicationinteraction engine 334 can send and/or receive data from variousapplications at the OEM vehicle computing device 318, and determinewhen, and/or how much data, to share with other applications at the OEMvehicle computing device 318. For example, a third-party application 332can be available at the OEM vehicle computing device 318, and thethird-party application 332 can be in communication with a third-partyserver device 350. The third-party server device 350 can processrequests provided via the OEM vehicle computing device 318 using aninput request engine 354, and can be responsive to the requests using acommand engine 356.

In some implementations, an application status engine 352 of thethird-party server device 350 can generate server data 358, which cancharacterize a status of one or more third-party applications 332.Furthermore, because a third-party application 332 can be associatedwith an account of a user, and the account can also be associated withthe automated assistant 304 (with prior permission from the user), datacan be shared between the third-party server device 350 and the serverdevice 302. For instance, changes in application status determined bythe application status engine 352, and/or operations being performed byone or more applications, can be communicated to the automated assistant304, in order to put the automated assistant 304 on notice of one ormore actions being performed by the third-party application(s) 332. Athird-party interaction engine 326 can receive data from the third-partyserver device 350, the OEM vehicle computing device 318, a clientdevice, and/or any other device or apparatus, in order to providecontextual data for the automated assistant 304.

A suggestion engine 328 of the automated assistant 304 can use thecontextual data in order to generate suggestions regarding relatedactions that can be initialized by the automated assistant 304.Furthermore, suggestion data characterizing the suggestions can bestored as assistant data 316 and processed by a suggestion rankingengine 336. The suggestion ranking engine 336 can rank suggestions atleast partially based on the contextual data. Furthermore, in someimplementations, the suggestion ranking engine 336 can be ranked, usingthe contextual data, in order to prevent suggesting actions that haverecently been performed by an application, and/or are currently beingperformed by an application.

In some implementations, suggestions can be ranked and/or presentedaccording to the contextual data, which can include details of a routeof the user. For instance, the contextual data can indicate whether theuser is currently being driven on a portion of the route that lasts forX minutes and/or Y miles. In some implementations, when the portion ofthe route satisfies a time and/or distance threshold, the suggestionengine 328 can cause one or more suggestion elements to be presented ata graphical user interface of the OEM vehicle computing device 318.However, when the portion of the route does not satisfy the time and/orthe distance threshold, the suggestion engine 328 can bypass causing thegraphical user interface to present one or more suggestion elements.

FIG. 4 illustrates a method 400 for providing one or more suggestionelements at a graphical user interface of a third-party vehiclecomputing device and/or a third-party application operating at thethird-party vehicle computing device. The method 400 can be performed byone or more computing devices, applications, and/or any other apparatusor module capable of providing suggestions at a user interface. Themethod 400 can include an operation 402 that includes receiving vehicleapplication data characterizing content that is being rendered at adisplay panel of a vehicle computing device. The vehicle computingdevice can operate according to limitations that restrict localcommunication between applications that are accessible via the vehiclecomputing device.

For example, the vehicle computing device can include an OEM third-partyvehicle application, one or more other third-party applications, and/oran automated assistant. The automated assistant can be remote from thevehicle computing device. Therefore, the vehicle computing device canlimit communications between the automated assistant and the one or moreother third-party applications. For instance, when the vehicle computingdevice includes a third-party navigation application and/or athird-party messaging application, the aforementioned applications maybe restricted from directly communicating with the automated assistant,and instead restricted to communicating exclusively via the vehiclecomputing device, optionally through the vehicle application. However,the OEM vehicle application can generate the vehicle application data,which can characterized or indicate content that is being rendered at adisplay panel of the vehicle computing device. Alternatively, oradditionally, the OEM vehicle application can generate vehicleapplication data characterizing one or more operations being performedby one or more other third-party applications, the vehicle, and/or anyother apparatus or module that can be associated with the vehicle and/orthe vehicle computing device. Restricting direct communication accessbetween an automated assistant and other third-party applications canhelp to maintain the integrity of the vehicle systems, which can beimportant for vehicle safety and reliability.

The method 400 can further include an operation 404 of determining arelationship between the received vehicle application data and assistantdata that is associated with the automated assistant. For example, theoperation 404 can include determining whether the vehicle applicationdata characterizes any information that has been the subject of one ormore dialogue sessions between the user and the automated assistant.Alternatively, or additionally, the operation 404 can includedetermining whether the vehicle application data characterizes anyinformation that has been accessed by the user using one or morecomputing devices that provide access to the automated assistant. Accessto such information can be provided with permission from the user, inorder to protect information that the user does not want other devicesand/or persons to access. Alternatively, or additionally, the operation404 can further include determining whether the vehicle application dataand the assistant data characterize information that can be associatedwith a type of data and/or subjective information. For example, arelationship between the vehicle application data and the assistant datacan be identified when the vehicle application data indicates that anincoming call is being received at the vehicle computing device from aparticular person, and that the automated assistant data indicates thatthe user has previously invoked the automated assistant to answer callsfrom the same particular person.

The method 400 can further include an optional operation 406 ofdetermining whether the (vehicle) application data characterizes a pastaction(s). A past action can be an action (e.g., providing anotification, sending data, accessing data, generating data, etc.) Thatthat has been executed within a period of time before the vehicleapplication data was received. In some implementations, the period oftime can be a threshold period of time that is static or dynamicaccording to properties of interactions between the user and of thevehicle computing device, and/or the user and the automated assistant.As an example, a third-party application available at the vehiclecomputing device can provide a notification indicating that a fluidsensor of the vehicle is in an alarm state. This notification can beconsidered a past action because the notification is being provided atthe graphical user interface rendered at the display panel within athreshold period of time of receiving the vehicle application data. Insome implementations, a time at which the notification was presented canbe determined based on time information that is characterized by thevehicle application data. The automated assistant can have access to anapplication for tracking certain properties of the vehicle, andtherefore can also be privy to the information characterized by thenotification about the fluid sensor. However, depending on when thenotification was presented at the display panel of the vehicle computingdevice, the automated assistant may or may not bypass causing anotification about the fluid sensor to appear at the display panel.

When the application data characterizes a past action at the operation406, the method 400 can proceed to the operation 408, which can includegenerating other suggestion data that avoids suggesting the past action.As an example, when the third-party application provides a notificationrelated to the fluid sensor, the other suggestion data can avoidsuggesting obtaining information about the status of the fluid sensor byproviding one or more suggestions that do not include requesting theinformation detailing the status of the fluid sensor. Rather, theautomated assistant can generate suggestion data characterizing a spokenutterance for requesting information about purchasing the fluidcorresponding to the fluid sensor, sending a message that includes theinformation detailing the status of the fluid sensor, placing a phonecall to a business that performs maintenance associated with the fluidsensor, and/or any other request that is different from requesting theinformation detailing the status of the fluid sensor.

When the application data does not characterize a past action asdetermined at the operation 406, the method 400 can proceed to theoperation 410. The operation 410 can include generating suggestion databased on the relationship between the vehicle application data and theassistant data. For example, when a graphical user interface at thethird-party application includes content characterizing a route to aparticular destination, and the automated assistant has not provided anotification related to the fluid sensor within a threshold period oftime, the automated assistant can provide the suggestion datacorresponding to a notification for the fluid sensor. The suggestiondata can characterize natural language content, such as a spokenutterance that, when spoken by the user, causes the automated assistantto provide information related to a status of the fluid sensor. Forexample, the spoken utterance can be, “Assistant, what is the status ofthe fluid sensor?”

The method 400 can proceed from the operation 410 and/or the operation408 to the operation 412, which can include determining a priority forthe suggestion data and/or the other suggestion data based on therelationship between the vehicle application data and the assistantdata. In some implementations, the operation 412 can be an optionaloperation. In some implementations, a strength of correlation betweenthe beautiful vehicle application data and the assistant data can beused to determine the priority for the suggestion data. For instance, acorrelation for vehicle application data that characterizes naturallanguage content explicitly included in a dialogue session between theautomated assistant and the user can be characterized as having astronger correlation then another correlation between the vehicleapplication data and the assistant data when a past dialogue sessionbetween the user and the automated assistant resulted in an action thatis characterized by the natural language content.

The method 400 can proceed from the operation 412 to the operation 414,which can include causing congestion suggestion element to be renderedby the third-party application based on the suggestion data. Optionally,the suggestion element can be rendered by the vehicle application.Causing the suggestion element to be rendered by the vehicle applicationcan comprise providing action data from the automated assistant to thevehicle application. The action data can be indicative of an action tobe performed by the vehicle application upon selection of the suggestionelement, and can comprise, for example, parameters for execution of anaction associated with the suggestion element. The suggestion elementcan be a selectable element that can be selected by the user tapping onthe display panel at a location where the suggestion element ispresented, and/or by the user providing a spoken utterance that includesat least some amount of the natural language content included in thesuggestion element. When the suggestion element is rendered at thedisplay panel, the method 400 can proceed to an optional operation 416of determining whether the rendered suggestion element has been selectedwithin a particular period of time. In some embodiments, the actionassociated with the suggestion element can be performed by one of theOEM vehicle applications or the third party application based on theaction data. Causing the native OEM vehicle application to perform theaction can improve security in instances where it is undesirable forthird party applications to have access to critical vehicle systems, forexample, in autonomous vehicles access to the sensor systems ornavigation systems may be restricted to native applications of thevehicle. In other embodiments, the action data can comprise a unique IDnumber provided by the automated assistant; selection of the suggestionelement by a user can cause the vehicle application to cause theautomated assistant to perform the action associated with the unique IDnumber of the action data of the suggestion element. Causing theautomated assistant to perform the application can improve the privacyof a user, since third party applications or native OEM applications ofthe vehicle do not have access to information about the user or aboutthe actions performed by the automated assistant based on thecorrelation or association between the vehicle application data and theassistant data. Security may therefore be improved for the user.

When they rendered suggestion element is selected within a particularperiod of time, the method can proceed from the operation 416 to theoptional operation 418. The optional operation 418 can include assigningpriority for the suggestion data according to the selection of thesuggestion. However, if the user does not select the suggestion within aparticular period of time, the method can proceed from the operation 416to the operation 420. The operation 420 can include assigning priorityfor the suggestion data according to the non-selection of the suggestionelement. In some implementations, a priority of the suggestion data canbe increased in response to the user selecting the suggestion elementand/or the priority of the suggestion data can be decreased in responseto the user not selecting the suggestion element within the particularperiod of time. In this way, as the user interacts with the vehiclecomputing device, suggestions can be ranked according to their relevanceto the user, as well as whether the user has already acknowledgedcontent corresponding to the suggestion data. The method 400 can cycleaccording to new vehicle application data being rendered by thethird-party vehicle application, in order to promote various functionsat the vehicle and/or the vehicle computing device, thereby alsopromoting more efficient uses of the vehicle and/or the vehiclecomputing device.

FIG. 5 is a block diagram of an example computer system 510. Computersystem 510 typically includes at least one processor 514 whichcommunicates with a number of peripheral devices via bus subsystem 512.These peripheral devices may include a storage subsystem 524, including,for example, a memory 525 and a file storage subsystem 526, userinterface output devices 520, user interface input devices 522, and anetwork interface subsystem 516. The input and output devices allow userinteraction with computer system 510. Network interface subsystem 516provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 522 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 510 or onto a communication network.

User interface output devices 520 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 510 to the user or to another machine or computersystem.

Storage subsystem 524 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 524 may include the logic toperform selected aspects of method 400, and/or to implement one or moreof OEM vehicle computing device 104 and/or 204, OEM vehicle applications124 and/or 224, third-party applications 126 and/or 226, automatedassistant 128 and/or 228, vehicle 108 and/or 208, server device 302, OEMvehicle computing device 318, third-party server device 350, and/or anyother application, device, apparatus, and/or module discussed herein.

These software modules are generally executed by processor 514 alone orin combination with other processors. Memory 525 used in the storagesubsystem 524 can include a number of memories including a main randomaccess memory (RAM) 530 for storage of instructions and data duringprogram execution and a read only memory (ROM) 532 in which fixedinstructions are stored. A file storage subsystem 526 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 526 in the storage subsystem 524, or inother machines accessible by the processor(s) 514.

Bus subsystem 512 provides a mechanism for letting the variouscomponents and subsystems of computer system 510 communicate with eachother as intended. Although bus subsystem 512 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 510 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 510depicted in FIG. 5 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 510 are possible having more or fewer components thanthe computer system depicted in FIG. 5.

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore it is stored or used, so that personal identifiable informationis removed. For example, a user's identity may be treated so that nopersonal identifiable information can be determined for the user, or auser's geographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

In some implementations, a method implemented by one or more processorsis set forth as including operations such as receiving, via a vehicleapplication that is operational when a user is riding in a vehicle,vehicle application data that corresponds to content being rendered at agraphical user interface of the vehicle application, wherein thegraphical user interface is displayed at a display panel that is incommunication with a vehicle computing device of the vehicle. The methodcan further include determining, in response to receiving the vehicleapplication data, that the vehicle application data is associated withassistant data that is available via an automated assistant, wherein theautomated assistant is accessible via the vehicle computing device. Themethod can further include generating suggestion data in response todetermining that the vehicle application data is associated with theassistant data, wherein the suggestion data identifies an action that isassociated with the assistant data and the vehicle application data, andthe suggestion data characterizes natural language content that isdifferent from the content that is being rendered at the display panel.The method can further include determining, based on the suggestiondata, a priority of the suggestion data relative to priority data thatis associated with previously generated suggestion data, wherein thepreviously generated suggestion data corresponds to one or moresuggestion elements previously rendered at the graphical user interfaceof the vehicle application. The method can further include, when thepriority of the suggestion data relative to the priority data indicatesthat the suggestion data is prioritized over the other suggestion data:causing a particular suggestion element to be rendered at the graphicaluser interface of the vehicle application based on the suggestion dataand while the user is riding in the vehicle, wherein the particularsuggestion element includes the natural language content.

In some implementations, the priority data is based on one or moreprevious instances in which the user and/or one or more other usersselected the one or more suggestion elements while the user and/or theone or more other users were riding in the vehicle. In someimplementations, the method can further include determining an estimatedtime of arrival of the user to a destination location, whereindetermining the priority of the suggestion data relative to the prioritydata that is associated with other suggestion data is at least partiallybased on the estimated time of arrival of the user to the destinationlocation. In some implementations, the method can further include, whenthe priority of the suggestion data relative to the priority dataindicates that the suggestion data is prioritized over the othersuggestion data: determining, subsequent to causing the particularsuggestion element to be rendered at the graphical user interface of thevehicle application, that the particular suggestion element wasselected, and causing, in response to determining that the particularsuggestion element was selected, the priority of the suggestion data tobe modified.

In some implementations, the graphical user interface of the vehicleapplication includes a first area that includes the content, and asecond area that includes other content, and wherein causing theparticular suggestion element to be rendered at the graphical userinterface of the vehicle application includes: causing the particularsuggestion element to be rendered at the first area of the graphicaluser interface and another suggestion element to be rendered at thesecond area of the graphical user interface, wherein the othersuggestion element is at least partially based on the other content. Insome implementations, the vehicle application data also corresponds tothe other content, and generating the suggestion data includesidentifying another action that is associated with the other content. Insome implementations, the method can further include generatingadditional suggestion data that is based on the content provided in thefirst area of the graphical user interface and on the other contentprovided in the second area of the graphical user interface, wherein theadditional suggestion data identifies an additional action that isassociated with the assistant data and the vehicle application data, andthe suggestion data characterizes other natural language content that isdifferent from the content and the other content.

In some implementations, the graphical user interface of the vehicleapplication include a third area corresponding to the automatedassistant, and the method further comprises: causing a separatesuggestion element to be rendered at the third area of the graphicaluser interface of the vehicle application based on the additionalsuggestion data and while the user is riding in the vehicle, wherein theseparate suggestion element includes the other natural language content.In some implementations, causing the particular suggestion element to berendered at the graphical user interface of the vehicle applicationbased on the suggestion data includes providing action data from theautomated assistant to the vehicle application. In some implementations,the particular suggestion element is rendered by the vehicleapplication, and the natural language content of the suggestion elementcharacterizes a spoken utterance that, when spoken by the user to thevehicle computing device, causes the action to be initialized via theautomated assistant. In some implementations, the method can furtherinclude, when the priority of the suggestion data relative to thepriority data indicates that the suggestion data is prioritized over theother suggestion data: determining that the user has provided a spokeninput that corresponds to the natural language content of the suggestionelement, and causing, in response to determining that the user hasprovided the spoken input, the automated assistant to initialize theaction, wherein the action is performed by a separate vehicleapplication in response to the automated assistant initializing theaction.

In some implementations, the automated assistant is provided by a sourcethat generated the priority data, the vehicle application data isgenerated by a third-party source that is different from the source, andthe separate vehicle application is provided by a separate source fromthe source and the third-party source. In some implementations, thevehicle application is a navigation application provided by the sourceand the separate vehicle application is a communication and/or mediaapplication provided by the separate source. In some implementations,the automated assistant is provided by a source that generated thepriority data, and the vehicle application data is generated by athird-party source that is different from the source. In someimplementations, the automated assistant is remote from the vehiclecomputing device of the vehicle.

In other implementations, a method implemented by one or more processorsis set forth as including operations such as determining, by a vehicleapplication of a vehicle computing device, that a third-partyapplication is providing content via a graphical user interface that isbeing rendered at a display panel of the vehicle computing device,wherein the vehicle application is accessible to a user via the vehiclecomputing device and the vehicle computing device is part of a vehiclethat is navigating the user to a destination. In some implementations,the method can further include generating, based on determining that thegraphical user interface includes the content, vehicle application datathat characterizes at least a portion of the content provided by thethird-party application. In some implementations, the method can furtherinclude providing, by the vehicle application of the vehicle computingdevice, the vehicle application data to an automated assistant, whereinthe automated assistant is also accessible to the user via the vehiclecomputing device. In some implementations, the method can furtherinclude causing, based on providing the vehicle application data to theautomated assistant, the automated assistant to provide suggestion datathat is generated based on a correspondence between assistant data andthe vehicle application data, wherein the assistant data is associatedwith an account of the user. In some implementations, the method canfurther include receiving, by the vehicle application and from theautomated assistant, the suggestion data, wherein the suggestion datacharacterizes natural language content that, when spoken by the user tothe vehicle computing device, causes the automated assistant toinitialize performance of an action by the third-party application. Insome implementations, the method can further include causing, inresponse to receiving the suggestion data from the automated assistant,the third-party application to render the natural language content atthe graphical user interface of the third-party application, wherein thenatural language content is rendered at the graphical user interfacesimultaneous to the content being rendered at the graphical userinterface, and the vehicle navigating the user to the destination.

In some implementations, the method can further include, subsequent tothe natural language content being rendered at the graphical userinterface: determining that the user and/or another user has provided aspoken utterance that includes at least some of the natural languagecontent rendered at the graphical user interface, and causing, inresponse to determining that the user and/or the other user has providedthe spoken utterance, the automated assistant to initialize performanceof an action by the third-party application. In some implementations,the method can further include determining, in response to receiving thesuggestion data, an amount of time and/or distance remaining for thevehicle to navigate the user to the destination; and determining whetherthe amount of time and/or the distance remaining for the vehicle tonavigate the user to the destination satisfies a threshold, whereincausing the third-party application to render the natural languagecontent at the graphical user interface of the third-party applicationis performed when the amount of time and/or the distance remainingsatisfies the threshold. In some implementations, the method can furtherinclude providing updated content data characterizing additional datathat is being presented at the graphical user interface.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as providing, froma third-party application to a vehicle application, content datacharacterizing content to be rendered at a display panel of a vehiclecomputing device, wherein the vehicle computing device is included in avehicle that is driving a user to a destination. The method can furtherinclude causing, based on providing the content data to the vehicleapplication, the vehicle application to provide at least a portion ofthe content data to an automated assistant, wherein the automatedassistant is accessible to the user via one or more interfaces of thevehicle computing device, and wherein the vehicle computing devicelimits accessibility of the content data to the automated assistant. Themethod can further include receiving, from the vehicle application,suggestion data that is at least partially generated based on theautomated assistant receiving at least the portion of the content data,wherein the suggestion data characterizes natural language content that,when included in a spoken utterance from the user to the automatedassistant, causes the automated assistant to initialize performance ofone or more actions by the third-party application, the vehicleapplication, and/or another third-party application. The method cancausing, in response to receiving the suggestion data from the vehicleapplication, a display panel of the vehicle computing to render thenatural language content at a graphical user interface. The method canfurther include when the user provides the spoken utterance to theautomated assistant via an interface of the vehicle computing device:causing the automated assistant to initialize performance of the one ormore actions by the third-party application, the vehicle application,and/or another third-party application.

In some implementations, at least the portion of the content data isselected for providing to the automated assistant based on rendered datathat is displayed at the display panel of the vehicle computing device.In some implementations, the method can further include, when the userprovides the spoken utterance to the automated assistant via aninterface of the vehicle computing device: providing, via a serverdevice that is in communication with the third-party application, anoperating status of the one or more actions to another computing devicethat is separate from the vehicle computing device.

We claim:
 1. A method implemented by one or more processors, the methodcomprising: receiving, via a vehicle application that is operationalwhen a user is riding in a vehicle, vehicle application data indicatinga destination location of the user, that corresponds to content beingrendered at a graphical user interface of the vehicle application, wherethe graphical user interface is displayed at a display panel that is incommunication with a vehicle computing device of the vehicle; receivingassistant data including a communication from an additional user, notriding in the vehicle, that is available via an automated assistant,where the automated assistant is accessible via the vehicle computingdevice; in response to receiving the assistant data and receiving thevehicle application data that indicates the destination location,determining an estimated time of arrival of the user to the destinationlocation; in response to determining the estimated time of arrival,generating suggestion data which identifies an action of transmittingthe estimated time of arrival to the additional user; and causing, whilethe user is riding in the vehicle and before the user arrives at thedestination location, a particular suggestion element to be rendered atthe graphical user interface of the vehicle application based on thesuggestion data.
 2. The method of claim 1, wherein generating thesuggestion data which identifies the action of transmitting theestimated time of arrival to the additional user comprises: generatingnatural language content identifying the action of transmitting theestimated time of arrival to the additional user.
 3. The method of claim2, wherein the natural language content of the particular suggestionelement characterizes a spoken utterance that, when spoken by the userto the vehicle computing device, causes the action to be initialized viathe automated assistant.
 4. The method of claim 3, subsequent to thenatural language content being rendered at the graphical user interface:determining the user and/or another user has provided the spokenutterance, and causing, in response to determining the user and/oranother user has provided the spoken utterance, the automated assistantagent to initialize performance of the action of transmitting theestimated time of arrival to the additional user.
 5. The method of claim1, wherein determining the estimated time of arrival of the user to thedestination location comprises determining an amount of time remainingfor the vehicle to navigate the user to the destination location.
 6. Themethod of claim 1, wherein determining the estimated time of arrival ofthe user to the destination location comprises determining a distanceremaining for the vehicle to navigate the user to the destinationlocation.
 7. The method of claim 1, wherein causing, while the user isriding in the vehicle and before the user arrives at the destinationlocation, the particular suggestion element to be rendered at thegraphical user interface of the vehicle application based on thesuggestion data includes: causing the particular suggestion element tobe rendered at a first area of the graphical user interface and anothersuggestion element to be rendered at a second area of the graphical userinterface, wherein the other suggestion element is at least partiallybased on other content.
 8. A system, comprising: one or more processors;and memory configured to store instructions that, when executed by theone or more processors, cause the one or more processors to performoperations that include: receiving, via a vehicle application that isoperational when a user is riding in a vehicle, vehicle application dataindicating a destination location of the user, that corresponds tocontent being rendered at a graphical user interface of the vehicleapplication, where the graphical user interface is displayed at adisplay panel that is in communication with a vehicle computing deviceof the vehicle; receiving assistant data including a communication froman additional user, not riding in the vehicle, that is available via anautomated assistant, where the automated assistant is accessible via thevehicle computing device; in response to receiving the assistant data,determining an estimated time of arrival of the user to the destinationlocation; in response to determining the estimated time of arrival,generating suggestion data which identifies an action of transmittingthe estimated time of arrival to the additional user; and causing, whilethe user is riding in the vehicle and before the user arrives at thedestination location, a particular suggestion element to be rendered atthe graphical user interface of the vehicle application based on thesuggestion data.
 9. The system of claim 8, wherein generating thesuggestion data which identifies the action of transmitting theestimated time of arrival to the additional user comprises: generatingnatural language content identifying the action of transmitting theestimated time of arrival to the additional user.
 10. The system ofclaim 9, wherein the natural language content of the particularsuggestion element characterizes a spoken utterance that, when spoken bythe user to the vehicle computing device, causes the action to beinitialized via the automated assistant.
 11. The system of claim 10,subsequent to the natural language content being rendered at thegraphical user interface, the operations further include: determiningthe user and/or another user has provided the spoken utterance, andcausing, in response to determining the user and/or another user hasprovided the spoken utterance, the automated assistant agent toinitialize performance of the action of transmitting the estimated timeof arrival to the additional user.
 12. The system of claim 8, whereindetermining the estimated time of arrival of the user to the destinationlocation comprises determining an amount of time remaining for thevehicle to navigate the user to the destination location.
 13. The systemof claim 8, wherein determining the estimated time of arrival of theuser to the destination location comprises determining a distanceremaining for the vehicle to navigate the user to the destinationlocation.
 14. The system of claim 8, wherein causing, while the user isriding in the vehicle and before the user arrives at the destinationlocation, the particular suggestion element to be rendered at thegraphical user interface of the vehicle application based on thesuggestion data includes: causing the particular suggestion element tobe rendered at a first area of the graphical user interface and anothersuggestion element to be rendered at a second area of the graphical userinterface, wherein the other suggestion element is at least partiallybased on other content.
 15. A non-transitory computer-readable mediumconfigured to store instructions that, when executed by one or moreprocessors, cause the one or more processors to perform operations thatinclude: receiving, via a vehicle application that is operational when auser is riding in a vehicle, vehicle application data indicating adestination location of the user, that corresponds to content beingrendered at a graphical user interface of the vehicle application, wherethe graphical user interface is displayed at a display panel that is incommunication with a vehicle computing device of the vehicle; receivingassistant data including a communication from an additional user, notriding in the vehicle, that is available via an automated assistant,where the automated assistant is accessible via the vehicle computingdevice; in response to receiving the assistant data, determining anestimated time of arrival of the user to the destination location; inresponse to determining the estimated time of arrival, generatingsuggestion data which identifies an action of transmitting the estimatedtime of arrival to the additional user; and causing, while the user isriding in the vehicle and before the user arrives at the destinationlocation, a particular suggestion element to be rendered at thegraphical user interface of the vehicle application based on thesuggestion data.
 16. The non-transitory computer-readable medium ofclaim 15, wherein generating the suggestion data which identifies theaction of transmitting the estimated time of arrival to the additionaluser comprises: generating natural language content identifying theaction of transmitting the estimated time of arrival to the additionaluser.
 17. The non-transitory computer-readable medium of claim 16,wherein the natural language content of the particular suggestionelement characterizes a spoken utterance that, when spoken by the userto the vehicle computing device, causes the action to be initialized viathe automated assistant.
 18. The non-transitory computer-readable mediumof claim 17, subsequent to the natural language content being renderedat the graphical user interface, the operations further include:determining the user and/or another user has provided the spokenutterance, and causing, in response to determining the user and/oranother user has provided the spoken utterance, the automated assistantagent to initialize performance of the action of transmitting theestimated time of arrival to the additional user.
 19. The non-transitorycomputer-readable medium of claim 15, wherein determining the estimatedtime of arrival of the user to the destination location comprisesdetermining an amount of time remaining for the vehicle to navigate theuser to the destination location.
 20. The non-transitorycomputer-readable medium of claim 15, wherein determining the estimatedtime of arrival of the user to the destination location comprisesdetermining a distance remaining for the vehicle to navigate the user tothe destination location.